Motion Saliency Map Generations for Video Data Analysis: Spatio-temporal Signatures in the Array Operations

Motion saliency map is a critical feature in rapid analysis of video data, especially in coping with visual information overload and cluttered background [1]. Its potential applications include automatic action recognition, recognition of moving objects, and target tracking [2, 3, 4]. Similar to the other feature saliency maps, a motion saliency map is a topographic representation of the motion salience at every location in the visual scene. The generation of a saliency map involves typically an intermediate representation of a hierarchical structure with multiple spatial scales, which is computationally intensive. Array data structures are a natural choice in the early processing stage at the fine scale levels, and processors supporting fast array operations in parallel are preferable. The generation of a motion saliency map for video data analysis is more computationally costly in memory space and data access or movement, in comparison to intensity, color, (edge) orientation saliency maps. The latter are generated from individual image frames while a motion saliency map generation involves multiple frames. Moreover, there are different approaches emerged in recent years for generating motion saliency maps, based on physical sensor models, neuronphysiological vision models [5, 6] or information-theoretic arguments [7, 5]. We describe three of the motion saliency map generation methods that are rich in array operations, which can be carried out rapidly on many-core processors such as the graphics processing units (GPUs). These methods differ in their spatial-temporal characteristics or signatures of the array operations. We show a wide range in the data locality and concurrency in each array operation and the data relay between array operations. In the following description and discussion we assume the simplified case that an image frame at time t in a video sequence is a square array I(x, y; t) of N pixels of grayscale values with two dimensional Cartesian coordinates (x, y).